AEGD: adaptive gradient descent with energy

We propose AEGD, a new algorithm for optimization of non-convex objective functions, based on dynamically updated 'energy' variable. The method is shown to be unconditionally energy stable, irrespective the base step size. prove energy-dependent convergence rates AEGD both and convex objectives, which suitably small size recovers desired batch gradient descent. also provide an bound stationary in stochastic setting. straightforward implement requires little tuning hyper-parameters. Experimental results demonstrate that works well large variety problems. Specifically, it robust with respect initial data, capable making rapid progress. shows comparable often better generalization performance than SGD momentum deep neural networks. code available at https://github.com/txping/AEGD.